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This  is  a  Special  Report  prepared  for  the  Air  Training  Command  ( ATC/XPTT  and  ATC/RSMD)  and  the  Air 
Force  Manpower  and  Personnel  Center  (AFMPC/MPCM). 


EXECUTIVE  SUMMARY 


Background 

Applicants  to  the  Air  Force  Officer  Training  School  (OTS)  are  currently  evaluated  by  a  selection  board  composed 
of  senior  Air  Force  officers.  Based  on  experience  with  the  board  selection  process,  concern  has  been  expressed  about 
possible  inconsistencies  in  judgment  among  members  on  the  same  board  and  among  selection  boards  conducted  at 
different  times.  This  work  was  accomplished  to  develop  a  systematic  procedure  to  overcome  these  difficulties,  as  outlined 
in  the  Bequest  for  Personnel  Research  (RPR)  79-03,  Development  of  a  General  Purpose  Selection  and  Classification 
Model  for  Officer  Accessions  in  the  Pipeline  Management  System. 


Findings 

A  computer-based  algorithm,  based  on  the  policy-specifying  technique,  was  developed  to  model  selection  board 
decisions.  A  group  of  14  policy  experts  was  assembled  to  develop  the  algorithm.  Thirteen  relevant  variables  were 
operationally  defined  and  combined  into  three  major  categories  (military  performance  measures,  civilian  performance 
measures,  objective  cognitive  measures).  To  resolve  problems  in  interpretation  of  grade  point  averages  (GPAs)  from 
different  academic  institutions,  a  supplementary  policy  algorithm  (the  Patton  rating)  was  developed  to  adjust  GPAs  based 
on  the  quality  of  the  institution. 

The  overall  value  of  the  various  combinations  of  attributes  was  expressed  in  the  form  of  a  100-point  OTS  payoff 
scale.  Every  5-point  increment  was  anchored  with  an  expected  behavior  characteristic  of  an  OTS  trainee  or  that  of  an 
active  duty  Air  Force  officer.  The  payoff  scale  was  used  at  each  stage  in  the  hierarchy  to  maintain  a  consistent  framework 
for  eliciting  policy  judgments. 

A  consensus  policy  was  developed  on  how  various  levels  on  the  13  variables  could  be  numerically  combined  to 
represent  an  overall  payoff  value  to  the  Air  Force.  The  resulting  model  was  checked  for  general  consistency  by  comparing 
the  applicant  group  ratings  as  determined  by  (a)  the  algorithm,  (b)  a  mock  selection  board  composed  of  policy  experts, 
and  (c)  an  actual  OTS  selection  panel. 


Conclusions 

1 .  The  selection  algorithm  developed  as  a  product  of  RPR  79-03  provides  a  reliable  and  clearly  specified  procedure 
for  rank-ordering  OTS  applicants.  This  algorithm  was  generally  consistent  with  mock  and  OTS  selection  board  procedures 
but  did  not  represent  an  exact  replication  of  either. 

2.  The  OTS  payoff  scale  provided  a  convenient  point  of  reference  for  developing  the  policy  model. 

3.  Procedures  developed  can  be  generalized  to  other  situations  where  a  single  “order  of  merit”  ranking  based  on 
multiple  input  variables  is  desired. 

Recommendation 


It  is  recommended  that  the  computer  algorithm  be  implemented  as  a  part  of  the  OTSapplicant  selection  system. 
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OFFICER  TRAINING  SCHOOL  SELECTION  ALGORITHM 


I.  BACKGROUND 

Applicants  for  Officer  Training  School  (OTS)  are  currently  evaluated  by  a  selection  board  composed  of  senior  Air 
Force  officers.  Decisions  about  who  is  best  qualified  for  entry  are  made  on  the  basis  of  a  review  of  the  applicant's  files, 
which  typically  include  identifying  data,  application  forms,  educational  transcripts,  letters  of  recommendation,  etc. 
Individual  board  membership  varies  over  time.  Eligible  officers  serve  on  the  board  on  a  voluntary  basis. 

Based  on  experience  with  the  board  selection  process  and  with  similar  systems  that  have  been  used  during  the  past 
in  Reserve  Officer  Training  Corps  and  Air  Force  Academy  screening,  concern  has  been  expressed  about  possible 
inconsistencies  in  judgment  among  members  on  the  same  board  and  among  selection  boards  conducted  at  different  times. 
The  main  problem  seems  to  stem  from  the  necessity  for  evaluating  an  overwhelming  amount  of  relevant  information  on 
each  applicant,  the  absence  of  clear-cut  standards,  and  the  subjective  nature  of  the  evaluation  process.  The  need  to 
develop  a  systematic  procedure  to  overcome  some  of  these  difficulties  was  documented  in  a  Request  for  Personnel 
Research  ( RPR  |  jointly  sponsored  by  the  Air  Training  Command  I  ATC|  and  the  Air  Force  Manpower  and  Personnel  Center 
(AFMPC)  (RPR  79-03,  Development  of  a  Cieneral  Purpose  Selection  and  Classification  Model  for  Officer  Accessions 
in  the  Pipeline  Management  System).  This  work  was  accomplished  in  support  of  that  requirement. 


It.  FINDINGS 

A  computer-fused  algorithm  was  develojieo  to  model  selection  hoard  decisions  based  on  the  policy-specifying 
technique  (Ward,  1977;  Ward,  Pina,  Fast,  &  Roberts,  1979).  Policy  specifying  requires  the  identification  of  policy 
experts,  selection  of  relevant  variables,  combination  of  variables  in  a  pair-wise  hierarchy,  and  assignment  of  payoff  values 
at  each  stage  of  the  hierarchy  representing  overall  benefit  to  the  Air  Force.  Since  the  objective  was  to  improve  OTS 
selection  tmard  procedures  rather  than  to  duplicate  them,  techniques  such  as  judgment  analysis  (Christal,  1968a)  or 
(mlicy  capturing  (Christal.  1968b)  were  not  considered  appropriate.  A  group  of  14  policy  experts  was  assembled  from 
various  offices  in  ATC  and  AFMPC  for  purposes  of  developing  the  algorithm.  Variables  appropriate  for  selecting  Air 
Force  officers  were  identified  via  questionnaires  completed  by  policy  experts.  Thirteen  relevant  variables  were 
o|M-raltonally  defined  and  combined  into  three  major  categories  to  serve  as  a  framework  for  decision  making.  The 
categories  were:  (a)  objective  cognitive  measures,  (bl  military  performance  measures,  and  (c)  civilian  performance 
measures  (figure  1).  The  variables  are  defined  in  more  detail  in  Appendix  B.  College  grade  point  average  (GPA) 
presented  special  problems  in  interpretation  liecause  of  the  known  variability  in  institutional  quality  (Astin,  1965; 
Bryden,  1981 ).  A  GPA  of  3.5  from  a  highly  selective  school  may  not  reflect  the  same  achievement  level  if  obtained  at 
a  less  selective  school.  A  supplementary  policy  algorithm,  termed  the  Patton  Rating,  was  developed  in  parallel  with 
the  selection  algorithm  to  “adjust”  GPAs  based  on  the  quality  of  the  institution  at  which  they  were  received.  Details 
o(  this  rating  system  arc  given  in  Appendix  (i. 

Ihc  overall  value  of  the  various  combinations  of  attributes  was  expressed  in  the  form  of  a  100-point  OTS  payoff 
scale.  Even  five-point  increment  was  anchored  with  an  expected  behavior  characteristic  of  an  OTS  trainee  or  that  of 
an  active  duty  Air  force  officer  as  shown  in  Appendix  A.  Anchors  were  developed  using  the  procedure  detailed  in 
(.amplH-ll,  Dunnetle.  Arvev.  anil  Hellervik  I  1973).  The  payoff  scale  was  used  at  each  stage  in  the  hierarchy  to  maintain 
a  consistent  framework  for  judgment. 

I  he  group  of  |Milicy  experts  met  in  weekly  3-hour  sessions  over  the  period  of  several  months  to  develop  a  consensus 
policy  on  the  selection  measures.  Agreement  was  reached  on  how  various  levels  on  the  1 3  variables  could  be  numerically 
combined  to  represent  an  overall  pavofl  value  to  the  Air  f  orce  (f  igure  1 ).  Most  ol  the  pair-wise  combinations  of  variables 
and  (unctions  arc  relatively  straightforward.  There  arc  two  switching  mechanisms  in  the  hierarchy.  The  first  (SI)  is  set 
to  reflect  whether  tin  applicant  has  recent  prior  military  service  data,  if  not,  information  from  the  recruiter  evaluation, 
nou-mihtarv  awards,  and  non-militarv  achievements  Is-eomes  the  primary  basis  lor  assessment.  If  the  applicant  had  prior 
military  service,  the  second  switch  i>Ul  is  set  according  on  the  amount  of  service.  The  policy  Imard  decided  that  the 
evaluation  of  a  person  »  mbitarv  performance  should  be  contingent  on  the  length  ol  prior  military  service,  f  or  a  given 
uppi icant.  onlv  one  of  the  follow  my  linn  lions  is  apubeif : 


1.  F07  ifOlo  4  years  military  service 

2.  F08  if  5  to  7  years 

3.  F09  if 8  to  15  years 


To  illustrate  the  operation  of  the  functions  in  the  hierarchy.  Table  1  shows  the  payoff  values  associated  with  all 
pair-wise  combinations  of  the  output  from  functions  F03  and  F12.  F03  summarizes  the  applicant’s  performance  on  the 
objective  cognitive  measures  (i.e.,  AFOQT  and  GPA),  and  F12  indicates  either  the  applicant’s  prior  military  performance 
or  participation/leadership  roles  in  civilian  life.  (See  discussion  of  SI  in  the  previous  paragraph.)  Function  F13,  shown 
in  the  table,  indicates  that  an  applicant  with  the  highest  level  of  performance  on  the  cognitive  measures  (F03  —  100) 
begins  with  a  final  payoff  score  of  30  if  F12  has  a  value  of  ^  15.  For  each  5-point  increase  in  F12,  the  applicant  gains 
approximately  4final  points  uptoamaximum of  lOOwhen  F12  =  100. 


Table  1 .  Final  Payoff  Matrix  for  the  OTS  Selection 
Algorithm  —  Function  13 
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After  the  complete  policy  had  been  modeled,  it  was  eheeked  for  general  consistency  by  comparing  the  applicant 
group  ratings  determined  bv  (a)  the  algorithm,  (b)  a  mock  selection  board  eom|>osed  of  policy  exports,  and  (e)  an  actual 
OTS  selection  panel.  Rank  order  correlations  among  the  three  scoring  systems  were  computed.  The  correlation  between 
the  algorithm  and  the  OTS  selection  Imard  was  .37.  ('  irrelation  between  the  algorithm's  ranking  and  that  of  the  mock 
Imard  was  .52  and  between  the  mock  board  and  the  OTS  panel,  .55. 

III.  CONCI.I  SIONS 

1.  The  selection  algorithm  developed  in  this  effort  provides  a  reliable  and  clearly  specified  procedure  for  rank- 
ordering  OTS  appl  ieants  based  on  consensus  judgments  of  Air  Force  |M>licy-makcrs. 

2.  The  computer-based  algorithm  is  generally  consistent  with  "mock"  and  “actual”  OTS  selection  board  procedures 
but  does  not  exactly1  replicate  either.  This  is  because  the  algorithm  incorporates  a  wide  variety  of  information  in  a  complex 
vet  consistent  manner,  thereby  eliminating  problems  with  instability  of  ratings  within  and  among  selection  boards. 

3.  The  ( ITS  pavoff  scale  provides  a  convenient  point  of  reference  for  developing  the  [ml  icy  model. 

IV.  APPLICATIONS 

The  computer  algorithm  can  be  applied  directly  to  selecting  OTS  applicants,  w  ith  only  a  modest  investment  in  clerical 
processing,  data  input  resources,  anti  a  small  microcomputer.  A  prudent  course  of  action  would  include  the  simultaneous 
use  of  regular  board  selection  procedures  cross-checked  over  a  period  ol  time  with  the  eorres|>ondiiig  selections  made 
bv  the  eouipmer  algorithm.  Any  obvious  and  unresolved  inconsistencies  would  serve  as  a  basis  for  further  refinement 
of  the  computer  algorithm.  Another  alternative  would  Is1  to  combine  the  computer  algorithm  score  with  that  produced 
bv  the  selection  board  into  a  higher-order  policy  model. 

Incorporation  of  the  computer-based  algorithm  into  til's  selection  procedures  would  Is-  expected  to  result  in  higher 
overall  quality  of  selectees  over  time  bv  minimizing  the  chance  that  lesser  qualified  applicants  would  be  selected  due 
In  inconsistencies  in  selection  procedures.  Renefil  would  also  accrue  by  having  Air  Force  selection  policy  stabilized 
mi  the  form  ol  the  algorithm. 

Instruments  and  procedures  used  in  this  effort  may  lie  ol  use  in  other  situations  where  a  single  “order  ol  merit" 
ranking  based  on  multiple,  and  possihlv  incommensurate,  input  variables  isdesired. 
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Table A-l.  OTS  Payoff  Scale 


Payoff 

Value  Anchor  Statement 

0  -  Hunks  out  of  OTS  immediately. 

5  -  Would  be  a  liability  to  the  Air  Force;  refuses  to  do  as  directed;  won’t  even  try. 

10  -  Has  no  initiative;  is  poor  performer;  has  little  motivation  and  no  leadership  characteristics; 

has  academic  difficulties  in  OTS  and  is  recycled. 

15  -  Has  no  initiative  or  creativity;  has  a  good  chance  of  failing  OTS. 

20  -  Has  occasional  poor  scores  in  OTS  academic  and  performance  evaluations. 

25  -  Is  reluctant  to  assume  any  leadership  responsibilities;  displays  initiative  infrequently. 

30  -  Would  not  accept  leadership  positions;  has  a  “why”  attitude,  but  would  graduate;  performs 

duties  with  encouragement  but  needs  help  in  training. 

35  -  Is  well  motivated  initially  upon  receiving  commission  but  drops  off  to  below-average 

motivation. 

VO  -  Meets  standards  but  needs  supervision. 

45  -  Is  motivated  for  some  but  not  all  aspects  of  assigned  tasks. 

50  -  Has  average  performance  at  OTS. 

55  -  Has  average  performance  in  academics  and  military  bearing  at  OTS. 

60  -  Shows  some  initiative;  would  finish  tasks  in  the  time  allotted;  should  be  able  to 

complete  OTS  and  most  technical  schools. 

65  -  Completes  assignments  to  best  ability;  helps  others  with  tasks. 

70  -  Has  good  leadership  potential. 

75  -  Frequently  outperforms  peers;  would  help  train  others  and  seek  extra  duties. 

80  -  Is  natural  performer  in  OTS,  almost  Distinguished  Graduate  (DG). 

85  -  Completes  training  rapidly  and  seeks  additional  information  about  tasks. 

90  -  Needs  no  supervision. 

95  -  Is  very  aggressive,  goal  oriented,  creative,  and  highly  motivated. 

100  -  Is  dynamic,  selfstarter:  Honor  Graduate. 
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Table  B-l.  OTS  Selection  Algorithm:  Variables  and  Definition* 


Variable  Name 


Definition 


A  FOQT  Verbal  (XOU  The  verbal  subtest  percentile  score  of  the  Air  Force  Officer  Qualifying  Test. 

AFOQT  Quant.  (X02)  The  quantitative  subtest  percentile  score  of  the  Air  Force  Officer  Qualifying  Test. 


GPA|X03) 


The  applicant's  undergraduate  college  grade  point  average  reported  on  a  four-point 
scale  (A  =  4,  B  =  3,C  =  2,  D  =  1). 


Patton  Rating (X04) 

Total  APR  Average  |X05| 

I-ast  three  APR  Average (X06) 
Military  Awards  (X07I 


Military  Achievements  (X08| 

Time  in  Service  (X09) 
Recruiter  Eval(X13) 

Non-Military  Awards  (XI 1) 


The  quality  index  of  the  applicant's  undergraduate  institution  as  a  function  of 
Scholastic  Aptitude  Test  (SAT),  Admission  Ratio,  Percept  Graduate  and 
Professional  Students,  and  Percent  of  Faculty  with  PhD  or  Professional  Degrees: 
measured  on  a  10-point  scale  ( 10  =  highest  rating)  and  reported  to  two 
significant  digits.  (See  Appendix  C) 

The  average  of  all  the  overall  Airman  Performance  Report  ratings  since 
applicant  enlisted. 

The  average  of  the  three  most  recent  overall  Airman  Performance  Report  ratings 
for  the  applicant. 

Military  decorations  and  awards  (e.g..  Medal  of  Honor,  Air  Force  Cross, 
Distinguished  Service  Cross,  Distinguished  Service  Medal,  Silver  Star,  Legion 
of  Merit,  Distinguished  Flying  Cross,  Airman’s  Medal,  Soldier’s  Medal,  Bronze 
Star  Medal.  Meritorious  Service  Medal,  Air  Medal,  Joint  Service  Commendation 
Medal,  Air  Force  Commendation  Medal,  Army  Commendation  Medal,  Purple  Heart, 
Outstanding  Airman  of  the  Year  Ribbon,  Air  Force  Achievement  Medal,  Basic 
Military  Trainee  Distinguished  Graduate  Ribbon).  Each  decoration  has  the  same 
point  value  as  reported  in  the  Weighted  Airman  Promotion  System  (WAPS)  portion 
of  the  Military  Training  Standard  (MTS)  Promotion  Fitness  Examination  (PFE) 

Study  Guide  ( A  FP  50-34). 

Includes  achievement  as  a  distinguished  graduate  of  a  military  training  program, 
selection  as  the  outstanding  airman  or  noncommissioned  officer  of  the  quarter  of 
the  year  for  a  wing  level  or  higher  organization.  Each  distinction  is  worth  one  point. 

The  total  active  Federal  military  service  reported  in  decimal  values. 

The  numerical  evaluation  provided  by  the  field  recruiting  officer  after  completing 
an  inverview  with  the  applicant.  Reported  through  the  100-point  OTS  payoff  scale. 

Includes  receipt  of  an  outstanding  employee  award,  any  award  or  honor  related  to 
a  civic  activity,  the  Outstanding  Young  Man  in  America  Award  or  the  Outstanding 
Young  Woman  in  America  Award,  and  membership  in  the  following  national  honor 
societies:  Alpha  Epsilon  Delta,  Alpha  Kappa  Delta,  Alpha  lambda  Delta,  Alpha 
Pi  Mu,  Chi  Epsilon,  Delta  Sigma,  Eta  Kappa  Nu.  Kappa  Delta  Pi,  Omega  Chi 
Epsilon,  Omicron  Delta  Epsilon,  Omicron  Nu,  Phi  Alpha  Theta,  Phi  Beta  Kappa, 

Phi  Eta  Sigma,  Phi  Kappa  Phi.  Pi  Delta  Phi,  Pi  Epsilon  Alpha.  Pi  Tau  Epsilon, 

Psi  Chi,  RhoChi,  Sigma  Delta  Pi,  Sigma  Gamma  Tau,  Sigma  Pi  Sigma,  Sigma  Tau 
Delta,  Tau  Beta  Pi,  and  Xi  Sigma  Pi.  Each  award  or  society  membership  is  worth 
one  point. 


Table  B- 1 .  ( Continued I 


Variable  Name 


Definition 


Non-Military  Achievements  (X 10)  Includes  holding  elective  office  to  any  of  the  following  organizations:  Sports 

Club;  Student  Judicial  System;  Living  Croup;  Clubs;  Professional  Organizations/ 
Societies;  Service  Organizations;  Religious  Organizations;  Organizations  Control 
Board;  Student  Government;  Fratemities/Sororities;  Debate  Team;  National  Honor 
Society;  National  Recognition  Societies;  the  Parent-Teachers  Association;  Sports 
league  Coach;  Religious  Youth  league;  Religious  Adult  Organization;  holding  a 
position  of  Scouting  leader  or  Director;  Varsity  Sports  Captain;  Varsity  Sports 
Manager;  Drill  Team  Captain;  Organizer  of  a  College  Political  Group  or  Campaign; 
Kditor  or  Feature  Writer  for  a  collegiate  paper,  annual,  magazine  or  anthology; 
Choir  Director;  Drill  Team  Member;  Member  of  Omicron  Delta  Kappa;  Member 
of  Lambda  Sigma;  Member  of  Molar  Board;  Campus  Radio  Station  Manager. 
Membership  in  the  following  national  recognition  societies:  Alpha  Kpsilon 
Rho.  Alpha  Fta  Rho,  Alpha  Phi  Omega.  Alpha  Psi  Omega,  Alpha  Zeta,  Block  and 
Bridle,  Delta  Sigma  Pi,  Gamma  Sigma  Sigma,  Kappa  Psi,  l.ambdaTau.  Omicron 
Kappa  Pi,  Phi  Chi  Theta.  Phi  Delta  Kappa,  Phi  Delta  Chi.  Phi  (.ambda  Kpsilon. 

Phi  Mu  Alpha,  Phi  Psi,  Phi  Zeta.  Pi  Alpha,  Pi  Mu  Kpsilon,  Sigma  Gamma 
Kpsilon,  Sigma  I  .ambda  Chi.  Kach  position  or  membership  counts  as  one  point. 

Kxtra-Curricular  Activities  (X 12)  Membership  in  the  following  organizations:  Sports  Club.  Living  Group,  (ilubs 

(other  than  a  sports  club),  professional  organizations/ societies,  service 
organizations,  religious  organizations,  organizations  control  hoard, 
fralernilies/sororities,  and  the  debate  team.  Kach  organizational  membership 
is  worth  one  point. 


Table  H-2.  OTS  Selection  Algorithm:  Equation* 

Function  Equation 

KOI  =  'to  -  o.ooi  iokiXoi  ,r»rt  o.<mh5.v,(\o2  7.5  r  o.ooooooh.'w>1(\oi  <)5r<\02  74r' 

F02  =  0.21X04  2oot  +■  (>..)i\oti  0.0025  t\o:i  2oot  i  \on 

HU  =  0.  15 1  Hll  i  +  0.55iK02i 

KOV  =  O.OOI25(\(K»  7 (K)r'  +  0.IKKHMKMMI25<X05-700r!(X06- 700r 

K05  =  55  +  I  5(\07i  +  2.05l\08)  -  0. 1  (X07)(X08| 

KOO  =  0.000001  <KOU'  O.OOOOOOOOO.lHKOfRM  '(F05  -  65)2 

K07  =  25  51X0*1  h -MUMM>4708|F06|2  ' +0.i  <04605 ( X00 -  1 M  FOOr  ' 

FOH  =  0.470UX0*)  7»’  +  0.002512 (F06)2  1  +  0.00002791  |X<W-7l'<F00r’  ' 

K(W  =  l(M)  0.«»475lXtW  7r  -  0.01  | K00 -  MM))2  +  O.IMKMW475|XO<>  7l2lKOO  KM)|-’ 

K10  =  55  +  1.5 1 X 10)  +  2. 5 1 X I  h  0. 1  (XIOiiXI  li 

FI  1  =  100  -  0.01  (X  14-  l(M)r’  -  0.04H8*>|F|0 -6.5|2  +  O.IHHHMBHH'HX  14  I00|2<FI0-05|2 

F12  =  15  +  1 . 154 (Kl  !• — 25 1  +  X 12  O.OI444(KU  25nXI2> 

F13  =  50  -  0.0000004(F05-HM)|*  +  O.K245(KI2  15*  0.<MMMMMMM>705«XF04  KM)|'|KI2  I5l 

ISnlf.  In  Function  Kl  2.  Kit  lakes  on  the  talue  <il  Kl  I  it  lilt*  a|i|>lu  aol  ha*  no  rt-1  fill  million  ser>  iit:  K07  I  the  applicant 
has()-  i  years  of  prior  service;  F08  if.’i-T  years;  or  HW  if  H  1,*  years. 


